A corpus of science journalism for analyzing writing quality

We introduce a corpus of science journalism articles, categorized in three levels of writing quality.1 The corpus fulfills a glaring need for realistic data on which applications concerned with predicting text quality can be developed and evaluated. In this article we describe how we identified, guided by the judgements of renowned journalists, samples of excellent, very good and typical writing. The first category comprises extraordinarily well-written pieces as identified by expert journalists. We expanded this set with other articles written by the authors of these excellent samples to form a set of very good writing samples. In addition, our corpus also comprises a larger set of typical journalistic writing. We provide details about the corpus and the text quality evaluations it can support. Our intention is to further extend the corpus with annotations of phenomena that reveal quantifiable differences between levels of writing quality. Here we introduce two such annotations that have promise for distinguishing amazing from typical writing: text generality/specificity and communicative goals. We present manual annotation experiments for specificity of text and also explore the feasibility of acquiring these annotations automatically. For communicative goals, we present an automatic clustering method to explore the possible set of communicative goals and develop guidelines for future manual annotations. We find that the annotation of general/specific nature on sentence level can be performed reasonably accurately fully automatically, while automatic annotations of communicative goals reveals salient characteristics of journalistic writing but does not align with categories we wish to annotate in future work. Still with the current automatic annotations, we provide evidence that features based on specificity and communicative goals are indeed predictive of writing quality.